System, method and mobile unit to sense objects or text and retrieve related information

ABSTRACT

A system, method and mobile unit to sense objects and/or text and retrieve related images, text or translations. The system may allow users to instantly access information about an object or text with a mobile device, such as a mobile phone, pager, personal digital assistant (PDA) or lap top computer.

CLAIM OF PRIORITY UNDER 35 USC

The present application claims priority to co-owned U.S. ProvisionalApplication No. 60/616,256, filed on Oct. 7, 2004, entitled “SYSTEM,METHOD AND MOBILE UNIT TO SENSE OBJECTS OR TEXT AND RETRIEVE IMAGES,TEXT OR TRANSLATIONS,” and is hereby incorporated by reference in itsentirety.

BACKGROUND

Some mobile phones currently have built-in cameras, which may be calledcamera phones. These camera phones allow a user to transmit images toother devices through a communication system.

SUMMARY

A system, method and mobile unit are described to sense objects and/ortext and retrieve information related to the object or text, such asimages, video, text or language translations. The system may allow usersto instantly view or access information about an object or text with amobile device, such as a mobile phone, pager, personal digital assistant(PDA) or lap top computer.

The system may be faster and more accurate than a system that only useshuman searchers to find information related to an object or text. Thesystem may use human searchers in addition to computer searchers.

Using images to identify objects or text may be faster and more accuratethan a system that uses speech-recognition to describe and identifyimages or text because images may be more precise than words. As thesaying goes, a picture may be better than a thousand words.

DESCRIPTION OF FIGURES

FIG. 1 illustrates a mobile unit, a communication system, a server, theInternet and storage units.

FIG. 2 illustrates a method of using the mobile unit, communicationsystem and server of FIG. 1.

FIG. 3 illustrates a method of using the mobile unit of FIG. 1.

FIG. 4 illustrates an example of information that may be displayed onthe mobile unit of FIG. 1.

DETAILED DESCRIPTION

Communication System

The communication system or network 140 may comprise any type ofcomponents configured to provide wireless communication with one or moremobile units 100. For example, the communication system 140 one or morebase stations (BS) 140 (also called base station transceivers), basestation controllers (BSCS) 142, mobile switching centers (MSCs) 143,packet data switch networks (PDSNs) 144, Internet servers 145, storageunits, etc. The communication system 140 may use a multiple accesstechnology, such as time division multiple access (TDMA), code divisionmultiple access (CDMA), wideband CDMA (WCDMA), Global System for MobileCommunications (GSM), time-division synchronous CDMA (TD-SCDMA), etc.Alternatively, the communication system 140 may comprise “hotspots”(transceivers) for WiFi or WiMax. The communication system 140 may usecommunication protocols, such as packet data transmission protocols.

Mobile Unit

The mobile unit 100 may be a mobile phone, a pager, a personal digitalassistant (PDA), a laptop computer, a digital camera or some othermobile device. The mobile unit 100 may be a commercially availabledevice, such as a camera phone made by Nokia, Motorola, Samsung, LGElectronics, Ericsson, Kyocera, etc. Alternatively, the mobile unit 100may be a modified commercially available device, e.g., a camera phonewith added software 107. Alternatively, the mobile unit 100 may be a newmobile device designed to implement some or all of the functionsdescribed herein.

The mobile unit 100 may comprise a sensor 102, a processor 104, a memory106, software 107, a database 116 of information, such as images,videos, text files and other data, a transceiver 108, a speaker 110, amicrophone 112, a display 114, a keypad or other user input component116, and an optional position location unit 118. The mobile unit 100 mayinclude other components in addition to or instead of the componentslisted herein. The mobile unit 100 may include less than all of thecomponents shown in FIG. 1. Some of the components in FIG. 1 may beintegrated.

The keypad 116 may include one or more keys or buttons on various sides(front, rear, top, back, lateral, etc.) of the mobile unit 100 and arenot limited to any one side. The keys or buttons are not limited to anymaterial, arrangement or structure.

The sensor 102 may take or capture a snapshot image (picture) or a videoof an object or text. The sensor 102 may be similar to standard imagesensors in currently available camera phones, or the sensor 102 may be aspecially designed image sensor. The sensor 102 may have aphotodetector, such as a charge-coupled device (CCD) and/orcomplementary metal oxide semiconductor (CMOS) circuit. The sensor 102may have a user-controlled zoom function or image magnification with oneof the keys of the keypad 116. Alternatively, the zoom function or imagemagnification may be automatically controlled by the mobile unit 100.The sensor 102 may have a user-controlled or automatic focus function tofocus on one or more objects.

User Commands—by Voice or Keypad

For any user-initiated functions on the mobile unit 100 describedherein, the user may press one or more keys on the keypad 116 or speak acommand into the microphone 112. For example, the user may press one ormore keys on the keypad 116 and/or speak a command into the microphone112 to activate the sensor 102, which senses or captures an image orpicture of an object or text.

Examples of objects may include a car, a building, a monument, a person,an animal, a plant, a piece of clothing, jewelry, a poster, a billboard,a sign, an image on a TV, computer display or other display, a comicbook, a stamp, a coin or other collectible. Examples of people includeprofessional athletes, sports teams, musicians, actors/actresses,celebrities, politicians, etc.

Examples of text may include a sign, an address, a business card, abook, billboard, poster, etc. The sensor 102 and/or the processor 104may convert the image of an object or text into data, such as digitaldata, e.g., a sequence of bits.

Before, during or after the sensor 102 is activated to capture an imageof an object or text, the user may enter one or more commands via one ormore keys on the keypad 116, the display 114 or microphone 112. Examplesof user commands may include “find information related to object,” “findinfo,” “find definition,” “find photo,” “find video,” “find relatedimages,” “find news,” “find history,” “find story,” “find product,”“buy,” “sell,” “find best deal,” “find nearest store,” “find seller,”“find person,” “find web site,” “find email,” “find address,” “findphone number,” “find nearest location,” “get map,” “find directions,”“translate,” and “translate in <language X>.”

The user may enter at least one search word via one or more keys on thekeypad 116 or say a word or phrase into the microphone 112. The searchword(s) may help identify the object and/or narrow a search to findspecific information related to the object. For example, the search wordmay be a name of the object or what the user believes is a name or briefdescription of the object.

In one configuration, the processor 104 may execute software 107 tosearch the database 116 and identify the object or text, retrieverelated information, and display information on the display 114 or playaudio files by the speaker 110.

In another configuration, the processor 104 may cause the transceiver108 to transmit the image data and a user command to the server 150 viathe communication system 140. The user command may be represented by ashort sequence of bits. This configuration may save processing power andmemory space on the mobile unit 100 compared to the configurationdescribed above. The server 150 may process the command and sendrequested data to the mobile unit 100.

In another configuration, the processor 104 first executes software 107to search the database 116 and try identifying the object or text. Ifthe object or text cannot be identified, e.g., within a time period, theprocessor 104 then causes the transceiver 108 to transmit the image dataand user command to the communication system 140.

The mobile unit 100 may automatically or upon a user input (e.g., usermay dial a phone number or press one or more buttons on the keypad 116or microphone 112) send the image data and user command to thecommunication system 140, which sends the image data and user command tothe server 150. For example, the mobile unit 100 may send the image datato a base station 141, which sends the data to a base station controller(BSC) 142, which sends the data to a mobile switching center (MSC) 143,which sends the data to a packet data switching network (PDSN) 144,which sends the data to the server 150. Other communication routesbetween the mobile unit 100 and server 150 may involve more components,less components, or other components of the communication system 140.

Display

FIG. 4 illustrates an example of some information that may be displayedon the display 114. All the information may not be displayed at one timeon one screen. Some of the information may be displayed on differentscreens of the display 114. The screens may be switched or scrolled byuser commands. The display 114 may display one or more pieces ofinformation according to when user commands are received by the mobileunit 100. For example, the display 114 may show an image 402 when theuser activates the sensor 102. The display 114 may show a search word ifthe user enters a search word. The display 114 may show an object nameand/or related image or video 404 after a search occurred for the image402.

As shown in FIG. 4, the display 114 may show information “options” (alsocalled categories or types) related to the object, such as buy, sell,recent news, history or background, statistics, etc. These options maybe displayed before a search and/or after a search. Displaying theoptions before a search may allow the user to select one or morecategories of information and help narrow a search before the searchoccurs. A narrower search may provide one or more advantages, such asreducing search time, processing power, processing time, memory at themobile unit 100 or server 150 to store search results, bandwidth and/orcapacity of the communication system 140, mobile unit 100 and/or server150, etc.

In addition to or instead of displaying these options before a search,the display 114 may show the options after a search occurs. Theseoptions allow the user to select one or more categories of informationto display or audio or video files to play after a search. These optionsafter a search may provide one or more advantages, such as reducingprocessing power, processing time, memory at the mobile unit 100 orserver 150 to store search results, bandwidth and/or capacity of thecommunication system 140, mobile unit 100 and/or server 150, etc.Without these options, a large amount of bandwidth and capacity may beconsumed to retrieve information for display or playback on the mobileunit 100.

The “buy” option may allow a user to buy an object in the image 402 or aproduct related to an object in the image 402. The buy option may belinked to a specific seller's web site or a web site such as Amazon.comor eBay. For example, if the image 402 shows a sports team, a musicianor movie, the buy option may retrieve product links related to the asports team, musician or movie, such as T-shirts, hats, other clothes,posters, compact discs, etc.

One or more options may be user-configurable or capable of beingmodified by the user. For example, the user may specify a local orspecific region for buying or selling an object in the image 402.Examples of regions may include states, such as Virginia, Maryland orCalifornia, east coast, west coast, midwest, south, or one or morecountries. The user may specify a time period for retrieving recent newssuch as within the last X days or Y weeks or Z months, etc. The user mayspecify a type of history, background or statistics. The order of theoptions displayed on the display 114 may also be configured or modifiedby the user.

As shown in FIG. 4, the user may enter or select the user's own optionor category of information to retrieve related to the image 402.

Server

The computer or server 150 may be a single unit or multiple units, suchas a distributed computer system. The server 150 may be at a singlelocation or multiple locations. The server 150 may be coupled to theInternet 180 and/or data storage units 190.

The server 150 may have software, hardware or a combination of softwareand hardware to implement the functions described herein. The server 150may include other components in addition to or instead of the componentslisted herein. Some or all of these components or functions may beimplemented in the mobile unit 100.

The server 150 may include a transceiver 151, processor 152, a memory154, a database 156, a search engine 158, an image processing module160, an object identifier 161, a text recognition module 162, a languagetranslation module 164, an information blocker 166, etc. These modulesare shown and described herein as separate modules, but one or more ofthese modules may be combined or integrated. Although shown in thememory 154, the search engine 158, image processing module 160, objectidentifier 161, text recognition module 162, language translation module164 and information blocker 166 may be implemented in whole or in partwith hardware, such as application specific integrated circuits (ASICs).

The server 150 may receive image data and one or more user commands fromthe mobile unit 100, retrieve related data/information from the Internet180 and/or data storage units 190, such as images, video and/or text(described below), and send the related data to the mobile unit 100 viathe communication system 140. The server 150 may perform these functionsin real time. The mobile unit 100 and server 150 may act as a quickdictionary, encyclopedia, reference source, or link to relatedinformation on the Internet 180.

The database 156 may store data, such as image files, video files and/ortext files, which may be transmitted by the transceiver 151 andcommunication system 140 to the mobile unit 100. The database 156 mayalso store text data in more than one language.

Search Engine

The search engine 158 may search for web sites, hypertext links, text,and other data or structures on the Internet 180 that match or arerelated to the image of text from the mobile unit 100. The search engine158 may be similar to search engines used by Yahoo, Google or a modifiedversion of such search engines.

Image Processing Module

The image processing module 160 may process the image from the mobileunit 100 to clarify an object or text. For example, the image processingmodule 160 may enhance or focus one or more parts of an image, filter orremove non-relevant objects (such as shadows, rain, snow), increase ordecrease contrast between light and dark areas, adjust brightness,adjust color, adjust focus, interpolate pixels to reduce the effects ofblurs, reflections, remove red eye, etc. Other image processingfunctions may be known to the camera, video and image processing art,but combining image processing and other functions described herein arenot known.

Object Identifier

The object identifier 161 may have one or more modules, which may beseparate or integrated. The object identifier 161 may try to identify orrecognize one or more objects of the image from the image processingmodule 160. The object identifier 161 may use the Internet 180 or thedatabase 156 to try to find objects or information (images, videos,text) that match or are similar to the object in the image from themobile unit 100.

The object identifier 161 may use object parameters such as shape (e.g.,round, rectangular, oval, angular, sharp edges, curved edges, humanface, shape of car, shape of commercial product), color, color pattern(e.g., monotone, spotted, striped, number of colors), surface texture(e.g., smooth, rough, reflective, furry, bumpy), material composition(e.g., metallic, wood, rock, plastic), arrangement, structure, facialfeatures, spacing between lines or regions, size of an object relativeto other objects, etc.

The object identifier 161 may use various levels of objectclassification to help identify an object. For example, one objectclassification may be whether the object is a living organism, such as aperson, animal, plant, fungi, etc. or a non-living organism, such as arock, vehicle, building, statue, sign, etc. The living organisms mayfurther be classified as a person, animal, plant or fungi. People may beclassified as professional athletes, sports teams, musicians,actors/actresses, celebrities, politicians, etc. The non-livingorganisms may further be classified as man-made or natural. Anotherobject classification may be whether the object is inanimate. Anotherobject classification may be whether the object is mechanical orelectrical.

Text Recognition Module

The text recognition module 162 may categorize text by language (e.g.,English, Chinese, Spanish) and style (e.g., cursive, font, numeric).Some text recognition techniques are known, such as characterrecognition machines available from Scantron Corporation of Irvine,Calif. Other text recognition techniques may be known to the camera,video and image processing art, but combining text recognition and otherfunctions described herein are not known.

Examples of Retrieved Images, Videos, Text, Audio Files

The server 150 may transmit some or all of the information related tothe original object or text to the mobile unit 100. To conservecommunication bandwidth and display space, the server 150 may only sendbrief titles of information found by the server 150.

The mobile unit 100 may display the information, such as text, images orvideo, on the display 114, as shown in FIG. 4. To conserve communicationbandwidth and display space, the mobile unit 100 may first display brieftitles of information found by the server 150. The mobile unit 100 mayprovide music or audible sounds that describe the object or text, suchas a voice recording defining, describing or translating the object ortext.

Retrieved images may include a file, a photograph, a drawing, acomputer-created graphic, a web site, a link to a website, a news storyor a commercial advertisement, which may be displayed on the mobiledevice 100.

Retrieved videos may include a documentary, a news clip, a music video,a biography, a cartoon, an advertisement, etc.

Retrieved text may be a definition, a description of the object, one ormore news articles, driving directions, geographic information, anadvertisement, a language translation, etc. The display 114 may displaya list of found categories of information or web site links, and theuser may scroll through the categories or links.

Translation

In one embodiment, the server 150 may provide a language translation ofthe text, for example, from Chinese to English. Any language may beavailable to be selected by the user of the mobile unit 100. The mobileunit 100 may allow a user to select a desired language or set a defaultlanguage for future translations.

Building a Database

The server 150 may build a database 156 of data (images, video files,audio files and text) related to objects and/or text by storingretrieved data from the Internet 180 and/or the storage units 190.

Information Blocker

A security or information blocker 166 may block certain information frombeing transmitted to the mobile unit 100 for privacy or safety issues.Examples may include names of non-famous people, personal phone numbers,email addresses, resident addresses, and other unauthorized information.

The functions described above may be implemented by hardware, softwareor a combination of hardware and software. Hardware may include aprocessor, a controller, application-specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), etc.

A System, Method and Mobile Unit with a User Identifier and a ScannableMedia to Store Credit or Debit Information

The mobile unit 100 may be a cellular phone, a personal digitalassistant (PDA), a laptop computer or some other mobile device.

The mobile unit 100 may include one or more ways to identify the user,such as a retina scanner, a fingerprint scanner, a voice scanner, aheart rate or pulse scanner, etc. For highest security, the mobile unit100 may have more than one way to identify the user. Alternatively, forcost or size concerns, the mobile unit 100 may have only one way toidentify the user.

The mobile unit 100 may require a user to identify himself or herselfusing one or more ways when the user activates the mobile unit 100 toperform one or more of the functions described below.

The mobile unit 100 may provide one or more functions of a person'swallet or purse or contents of a wallet or purse. For example, themobile unit 100 may serve as a credit card, a debit card, a driver'slicense, a passport membership card, an insurance card, anidentification card, a company security badge, a business card holder, amovie rental card, a library card, a health fitness membership card, apurchase receipt holder, etc. Thus, a user does not have to carry awallet.

Also, a user does not have to worry about the wallet being stolen. Ifthe mobile unit 100 is stolen, it is useless because the thief does nothave the retina, fingerprint and/or voice of the user to activate themobile unit 100.

The mobile unit 100 may have a medium to store credit, debit informationor other information listed above. The medium may be a scannable medium,such as a bar code or a magnetic strip. The medium may be an internalmedium, such as a memory and/or an integrated circuit, which isdetectable or adapted to communicate with external scanners orcommunicators, such as grocery store checkout scanners.

1. A mobile device comprising: a sensor configured to capture an imageof an object; a keypad configured to receive a user command to requesttitles of information of the object and select a category of informationof the object, the information comprising at least one of an image, avideo, an audio file, and text; a transceiver configured to (a)wirelessly transmit the captured image and the user command to acommunication system and (b) wirelessly receive titles of informationwithin the selected category of information from the communicationsystem; and a display configured to display the received titles.
 2. Themobile device of claim 1, wherein the mobile device comprises at leastone of a mobile phone, a laptop computer, a personal digital assistantand a camera.
 3. The mobile device of claim 1, wherein the categorycomprises at least one of a definition, a related image, a relatedvideo, news, history, purchase information, selling information,location, nearest store, person, web site, email, address, phone number,map, directions, and translation.
 4. The mobile device of claim 1,wherein the titles comprise at least one of links to picture files,links to video files, links to text files, and links to web sites. 5.The mobile device of claim 1, further comprising a voice recognitionunit configured to receive the user command.
 6. The mobile device ofclaim 5, wherein the voice recognition unit is configured to receive atleast one search word from the user, the search word being related tothe object.
 7. The mobile device of claim 1, wherein the keypad isconfigured to receive at least one search word from the user, the searchword being related to the object.
 8. The mobile device of claim 1,wherein the keypad is configured to receive a user command selecting atitle among the received titles, the display being configured to showinformation related to the selected title.
 9. The mobile device of claim1, wherein the display is configured to display a pre-stored list ofcategories for the user to select, the keypad being configured toreceive a user command to select one of the categories.
 10. The mobiledevice of claim 1, wherein the keypad is configured to receive a usercommand to configure a category.
 11. A method of operating a mobiledevice, the method comprising: capturing an image of an object with themobile device; receiving a user command to request titles of informationof the object and select a category of information of the object, theinformation comprising at least one of an image, a video, and text;wirelessly transmitting the captured image and the user command to acommunication system; wirelessly receiving titles of information withinthe selected category of information from the communication system; anddisplaying the received titles.
 12. The method of claim 11, furthercomprising receiving at least one search word from the user, the searchword being related to the object.
 13. A system comprising: acommunication network configured to receive a user command to findtitles of information for an object and data representing an image ofthe object from a mobile device; and a computer configured to (a)receive the user command and data from the communication network, (b)access at least one of a database and the Internet to find informationfor the object, the information comprising at least one of an image, avideo, an audio file, text, a hypertext link, and a web site, and (c)send titles of the found information to the communication network;wherein the communication network is configured to send the titles tothe mobile device.
 14. The system of claim 13, wherein the computer isconfigured to find a category of information, the category beingspecified in the user command.
 15. The system of claim 14, wherein thecategory comprises at least one of a definition, a related image, arelated video, news, history, purchase information, selling information,location, nearest store, person, web site, email, address, phone number,map, directions, and translation.
 16. The system of claim 13, whereinthe computer is configured to use a search word in the user command tofind information related to the object.
 17. The system of claim 13,wherein the computer is configured to use an Internet search engine tofind information related to the object.
 18. The system of claim 13,wherein the computer is further configured to (a) receive a second usercommand specifying one or more titles of found information and (b) sendfound information of the specified titles to the communication network.19. The system of claim 13, wherein the computer is further configuredto process the image to clarify the object.
 20. The system of claim 13,wherein the computer is configured to use parameters of the object tofind information related to the object, the parameters comprising atleast one of shape, features, color, color pattern, surface texture,material composition, arrangement, structure, spacing between lines, andsize of the object relative to other objects.
 21. The system of claim13, wherein the computer is configured to use a plurality of objectclassifications to find information related to the object.
 22. A methodcomprising: receiving a user command to find titles of information foran object and data representing an image of the object from a mobiledevice; accessing at least one of a database and the Internet to findinformation for the object, the information comprising at least one ofan image, a video, an audio file, text, a hypertext link, and a website; and sending titles of the found information to the mobile device.23. The method of claim 22, further comprising using parameters of theobject to find information related to the object, the parameterscomprising at least one of shape, features, color, color pattern,surface texture, material composition, arrangement, structure, spacingbetween lines, and size of the object relative to other objects.